Signal processing device and method, recording medium, and program

ABSTRACT

The present invention relates to a signal processing apparatus and a method, a recording medium, and a program, in which portions except an edge can be smoothed while the edge whose change in pixel value is steep is held correctly. A pixel of attention is determined in step S 11 , and a neighbouring pixel is determined in step S 12 . In step S 13 , a difference in pixel values between the pixel of attention and each neighbouring pixel is calculated. In step S 14 , according to a relationship in size between the difference and a threshold value ε., flags are raised for the neighbouring pixel and a neighbouring pixel which are symmetrical. Furthermore, a flag is also raised for a neighbouring pixel away from, in view of the pixel of attention, the symmetrical neighbouring pixel for which the flag is raised. In step S 15 , 7-pixel taps centered around the pixel of attention are averaged by weight. However, with respect to the neighbouring pixel for which the flag is raised, the pixel value is replaced by that of the pixel of attention C, and is calculated. The present invention can be applied to a video camera, a television receiver, etc.

TECHNICAL FIELD

The present invention relates to a signal processing apparatus and amethod, a recording medium, and a program, more particularly to a signalprocessing apparatus and a method, a recording medium, and a programwhich can be suitably used when emphasizing a texture of the insidesurrounded by edges without emphasizing the edges too much in an image,for example.

BACKGROUND ART

Conventionally, as for the video camera, a contrast emphasis method byway of gradation change, a high frequency component emphasis method ofemphasizing contrast of a high frequency component in an image, etc.have been proposed as a method of improving contrast (a differencebetween brightness and darkness) and a degree of sharpness (borderingprecision) of an image taken by imaging devices, such as CCD (ChargeCoupled Device), CMOS (Complementary Metal-Oxide Semiconductor), etc.

As the contrast emphasis method, a tone curve adjustment in which apixel level for each pixel of an image is converted with a function(hereafter referred to as a level conversion function) having apredetermined input-and-output relationship, and a method referred to ashistogram equalization in which the level conversion function isadaptively changed according to frequency distribution of pixel levelshave been proposed.

As the high frequency component emphasis method, a method referred to asan unsharp mask has been proposed that performs a so-called edgeenhancement in which an edge is extracted from the image and theextracted edge is emphasized.

In the contrast emphasis method, however, there is a problem in thatonly some bright regions within all the dynamic range (a differencebetween the maximum level and the minimum level) of the image can beincreased in contrast. In addition, there is another problem in that thecontrast is instead reduced in the brightest part and the darkest partof an image in the case of the tone curve adjustment, and near aluminescence region with low frequency distribution in the case of thehistogram equalization. Furthermore, in the high frequency componentemphasis method there is another problem in that only the contrast ofthe high frequency component of the image is emphasized, whereby theportion near the edges of the image is unnaturally emphasized, anddeterioration of image quality is unavoidable.

Then, there is a conventional method in which, in a situation whereedges having a steep change in a pixel value among input image data aresaved, portions other than the edges are amplified with an image signalprocessing apparatus constructed as shown in FIG. 1, to therebyemphasize the portions other than the edges (for example, JapanesePatent Application Publication (KOKAI) No. 2001-298621).

In the image signal processing apparatus as shown in FIG. 1, an inputtedimage signal is inputted into an ε filter 1 and a subtraction unit 2.The ε filter 1 receives, as an input, an image signal slightly changingon both sides of a steep edge as shown in FIG. 2A, converts it into animage signal in which only edges as shown in FIG. 2B are extracted, andwhich is outputted to the subtraction unit 2 and an adder 4.

Particular processing of the ε filter 1 will be described with referenceto FIGS. 3 and 4. The ε filter 1 determines each pixel of the inputimage to be a pixel of attention C one by one. As shown in FIG. 3, tapsare set up, including a plurality of neighbouring pixels (in this casesix pixels L3, L2, L1, R1, R2, R3) which are horizontally successive andcentered around the pixel of attention C. As shown in the followingexpression (1), pixel values of the pixel of attention C and theplurality of neighbouring pixels are subjected to weighted averaging bymeans of the tap coefficients (for example, {1, 2, 3, 4, 3, 2, 1}), andoutputted as a conversion result C′ corresponding to the pixel ofattention C.C′=(1×L3+2×L2+3×L1+4C+3×R1+2×R2+1×R3)/16   (1)

However, as shown in FIG. 4, a neighbouring pixel (in the case of FIG.4, neighbouring pixels R2 and R3) having a difference, which is largerthan a predetermined threshold value ε, between its pixel value and thepixel of attention C is calculated by replacing it by that of the pixelof attention C. That is, in the case of FIG. 4, the following expression(2) is calculated.C′=(1×L3+2×L2+3×L1+4×C+3×R1+2×C+1×C)/16   (2)

Now, returning to FIG. 1, the subtraction unit 2 subtracts the imagesignal inputted from the ε filter 1, from the image signal (the same asthe input to the ε filter 1) inputted from the preceding stage, extractsthe image signal slightly changing other than that of the edge, andoutputs it to an amplification unit 3. The amplification unit 3amplifies the output of the subtraction unit 2, and outputs it to theadder 4. The adder 4 adds the image signal in which parts other than theedge outputted from the amplification unit 3 are amplified, to the imagesignal in which only the edge inputted from the ε filter 1 is extracted.This produced total is the image signal in which the parts other thanthe edge are amplified in the situation where the steep edge is held.

Incidentally, in the ε filter 1 of the image signal processing apparatusas shown in FIG. 1, when an image signal having a larger edge size thanthe predetermined threshold value ε is inputted as shown in FIG. 5 forexample, the image signal after conversion is such that the phase isshifted to the left-hand side, as shown in FIG. 6. That is, there is aproblem in that the edge with the steep change in pixel value is notheld correctly, and an image quality may deteriorate.

DISCLOSURE OF THE INVENTION

The present invention is invented in view of such situations, and aimsto smooth portions except an edge while the edge whose change in pixelvalue is steep is held correctly.

A signal processing apparatus of the present invention includes:designation means for designating continuously arranged signals as asignal of attention one by one; determination means for determining apredetermined number of signals preceding the signal of attentiondesignated by the designation means, and a predetermined number ofsignals following the signal of attention, to be neighbouring signals;weight average means for averaging by weight the signal of attention andthe plurality of neighbouring signals; flag setting means forcalculating a difference in levels between the signal of attention and aneighbouring signal, judging whether or not the difference is largerthan a predetermined threshold value, and raising flags for theneighbouring signal and a neighbouring signal which are arrangedsymmetrically with respect to the signal of attention, when thedifference is judged to be larger than the predetermined thresholdvalue; and control means for controlling and causing the weightedaverage means to average by weight, using the signal of attentioninstead of the neighbouring signal for which the flag is raised.

On the basis of the position of the signal of attention, theabove-mentioned flag setting means can raise a flag also for theneighbouring signal arranged away from the neighbouring signal for whichthe flag has been raised.

The above-mentioned signal can be the pixel value of the pixel whichconstitutes the image.

A signal processing method of the present invention includes: adesignation step of designating continuously arranged signals as asignal of attention one by one; a determination step of determining apredetermined number of signals preceding the signal of attentiondesignated by way of the designation step, and a predetermined number ofsignals following the signal of attention, to be neighbouring signals; aweight average step of averaging by weight the signal of attention andthe plurality of neighbouring signals; a flag setting step ofcalculating a difference in levels between the signal of attention and aneighbouring signal, judging whether or not the difference is largerthan a predetermined threshold value, and raising flags for theneighbouring signal and a neighbouring signal which are arrangedsymmetrically with respect to the signal of attention, when thedifference is judged to be larger than the predetermined thresholdvalue; and a control step of controlling and causing a process in theweighted average step to average by weight, using the signal ofattention instead of the neighbouring signal for which the flag israised.

A program for a recording medium in accordance with the presentinvention includes: a designation step of designating continuouslyarranged signals as a signal of attention one by one; a determinationstep of determining a predetermined number of signals preceding thesignal of attention designated by way of the designation step, and apredetermined number of signals following the signal of attention, to beneighbouring signals; a weight average step of averaging by weight thesignal of attention and the plurality of neighbouring signals; a flagsetting step of calculating a difference in levels between the signal ofattention and a neighbouring signal, judging whether or not thedifference is larger than a predetermined threshold value, and raisingflags for the neighbouring signal and a neighbouring signal which arearranged symmetrically with respect to the signal of attention, when thedifference is judged to be larger than the predetermined thresholdvalue; and a control step of controlling and causing a process in theweighted average step to average by weight, using the signal ofattention instead of the neighbouring signal for which the flag israised.

A program in accordance with the present invention causes a computer toimplement processes including: a designation step of designatingcontinuously arranged signals as a signal of attention one by one; adetermination step of determining a predetermined number of signalspreceding the signal of attention designated by way of the designationstep, and a predetermined number of signals following the signal ofattention, to be neighbouring signals; a weight average step ofaveraging by weight the signal of attention and the plurality ofneighbouring signals; a flag setting step of calculating a difference inlevels between the signal of attention and a neighbouring signal,judging whether or not the difference is larger than a predeterminedthreshold value, and raising flags for the neighbouring signal and aneighbouring signal which are arranged symmetrically with respect to thesignal of attention, when the difference is judged to be larger than thepredetermined threshold value; and a control step of controlling andcausing a process in the weighted average step to average by weight,using the signal of attention instead of the neighbouring signal forwhich the flag is raised.

In the signal processing apparatus and the method, and the program inaccordance with the present invention, the difference in levels betweenthe signal of attention and the neighbouring signal is calculated, andit is judged whether or not the difference is larger than thepredetermined threshold value. When the difference is judged to belarger than the predetermined threshold value, flags are raised for theneighbouring signal and a neighbouring signal which are arrangedsymmetrically with respect to the signal of attention. In the case ofaveraging by weight the signal of attention and the plurality ofneighbouring signals, the signal of attention is used instead of theneighbouring signals for which the flags are raised.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of a structure of an imagesignal processing apparatus which emphasizes portions except an edgewhile a steep edge in an image is saved;

FIG. 2A is a graph showing an image signal inputted into and an imagesignal outputted from an ε filter of FIG. 1;

FIG. 2B is a graph showing an image signal inputted into and an imagesignal outputted from the ε filter of FIG. 1;

FIG. 3 is a diagram showing an example of taps used for the ε filter ofFIG. 1;

FIG. 4 is a chart for explaining operation of the ε filter of FIG. 1;

FIG. 5 is a chart showing an example of the image signal inputted intothe ε filter;

FIG. 6 is a chart showing an example of the image signal outputted fromthe ε filter, corresponding to the image signal as shown in FIG. 5;

FIG. 7 is a block diagram showing an example of a structure of anonlinear filter to which the present invention is applied;

FIG. 8 is a flow chart for explaining a first filtering process by meansof the nonlinear filter of FIG. 7;

FIG. 9 is a chart showing the image signal acquired by way of the firstfiltering process with the nonlinear filter, corresponding to the imagesignal as shown in FIG. 5;

FIG. 10 is a chart showing an example of the image signal inputted intothe nonlinear filter;

FIG. 11 is a chart showing an image signal acquired by way of the firstfiltering process with the nonlinear filter, corresponding to the imagesignal as shown in FIG. 10;

FIG. 12 is a flow chart for explaining a second filtering process bymeans of the nonlinear filter of FIG. 7;

FIG. 13 is a chart showing an image signal acquired by the secondfiltering process with the nonlinear filter, corresponding to the imagesignal as shown in FIG. 10; and

FIG. 14 is a block diagram showing an example of a structure of ageneral-purpose personal computer.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 3 shows an example of a structure of a nonlinear filter which is anembodiment of the present invention. This nonlinear filter 11 is used byreplacing the ε filter 1 of the image signal processing apparatus asshown in FIG. 1, and it is constituted by a control signal generatingunit 12 which generates a control signal based on an inputted imagesignal, and an LPF (Low Pass Filter) 13 which performs a filteringoperation according to a control signal generated by the control signalgenerating unit 12.

A first filtering process by means of the nonlinear filter 11 will bedescribed with reference to a flow chart of FIG. 8, referring to a casewhere 7-pixel taps, containing the pixel of attention C as shown in FIG.3, are used as an example.

In step S1, the control signal generating unit 12 determines each ofpixels in order of raster which constitute the inputted image signal, tobe the pixel of attention C one by one. In step S2, the control signalgenerating unit 12 determines the neighbouring pixels L3, L2, L1, R1,R2, and R3 which adjoin in the horizontal direction with respect to thepixel of attention C. In step S3, the control signal generating unit 12calculates a difference in pixel values between the pixel of attention Cand each of the neighbouring pixels L3, L2, L1, R1, R2, and R3.

In step S4, the control signal generating unit 12 judges whether or notthe difference calculated by way of the process in step S3 is largerthan the predetermined threshold value ε. Flags are raised for theneighbouring pixel judged to have the difference larger than thepredetermined threshold value ε, and a neighbouring pixel which arearranged symmetrically with respect to the pixel of attention C.

For example, when it is judged that the difference in pixel valuesbetween the pixel of attention C and the neighbouring pixel L1 is largerthan the predetermined threshold value ε, flags are raised for theneighbouring pixels L1 and R1. Similarly, when it is judged that thedifference in pixel values between the pixel of attention C and theneighbouring pixel R2 is larger than the predetermined threshold valueε, flags are raised for the neighbouring pixels R2 and L2.

Furthermore, in step S4, the control signal generating unit 12 outputs asignal indicative of whether or not there is a flag for the neighbouringpixels L3, L2, L1, R1, R2, and R3, to the LPF 13 as a control signal.

In step S5, the LPF 13 averages by weight the pixel of attention C andthe neighbouring pixels L3, L2, L1, R1, R2, and R3 by using expression(1), to be outputted as the conversion result C′ corresponding to thepixel of attention C. However, with respect to the neighbouring pixelfor which the flag is raised, the pixel value is replaced by that of thepixel of attention C, and is calculated.

For example, when the flags are raised for the neighbouring pixels L2and R2, the following expression (3) is operated.C′=(1×L3+2×C+3×L1+4×C+3×R1+2×C+ 1×R3)/16   (3)

Now, the description of the first filtering process is completed.According to this first filtering process, as shown in FIG. 5, forexample, when the image signal with the edge larger in size than thepredetermined threshold value ε is inputted, the image signal afterconversion is, as shown in FIG. 9, not shifted in phase and the edgewith the steep change in pixel value is held correctly.

However, as shown in FIG. 10, for example, when an image signal with theedge larger in size than the predetermined threshold value ε isinputted, the edge with the steep change in pixel value, with respect tothe image signal after conversion, may not be held correctly even by wayof the first filtering process, as shown in FIG. 11.

Then, the nonlinear filter 11 is arranged to perform a second filteringprocess as will be described below. The second filtering process will bedescribed with reference to a flow chart of FIG. 12, referring to a casewhere the 7-pixel taps, containing the pixel of attention C as shown inFIG. 3, are used as an example.

In step S11, the control signal generating unit 12 determines each ofthe pixels in order of raster which constitute the inputted image signalto be the pixel of attention C one by one. In step S12, the controlsignal generating unit 12 determines the neighbouring pixels L3, L2, L1,R1, R2, and R3 which adjoin in the horizontal direction with respect tothe pixel of attention C. In step S13, the control signal generatingunit 12 calculates a difference in pixel values between the pixel ofattention C and each of the neighbouring pixels L3, L2, L1, R1, R2, andR3.

In step S14, the control signal generating unit 12 judges whether or notthe difference calculated by way of the process in step S13 is largerthan the predetermined threshold value ε. Flags are raised for theneighbouring pixel judged to have the difference larger than thepredetermined threshold value ε, and a neighbouring pixel, which arearranged symmetrically with respect to the pixel of attention C.Furthermore, the control signal generating unit 12 also raises a flagfor a neighbouring pixel away, in view of the pixel of attention C, fromthe right and left neighbouring pixels which are symmetrical withrespect to the pixel of attention C.

For example, when it is judged that the difference in pixel valuesbetween the pixel of attention C and the neighbouring pixel L2 is largerthan the predetermined threshold value ε, flags are raised for theneighbouring pixels L2 and R2. Furthermore, flags are also raised forthe neighbouring pixels L3 and R3 which are away from the neighbouringpixels L2 and R2 in view of the pixel of attention C.

Furthermore, in step S14, the control signal generating unit 12 outputsa signal indicative of whether or not there is a flag for theneighbouring pixels L3, L2, L1, R1, R2, and R3, to the LPF 13 as acontrol signal.

In step S15, the LPF 13 averages by weight the pixel of attention C andthe neighbouring pixels L3, L2, L1, R1, R2, and R3 by using expression(1), to be outputted as the conversion result C′ corresponding to thepixel of attention C. However, with respect to the neighbouring pixelfor which the flag is raised, the pixel value is replaced by that of thepixel of attention C, and is calculated.

For example, when the flags are raised for the neighbouring pixels L3,L2, R2, and R3 the following expression (4) is operated.C′=(1×C+2×C+3×L1+4×C+3×R1+2×C+1×C)/16   (4)

Now, the description of the second filtering process is completed. Alsoaccording to this second filtering process, as shown in FIG. 5, forexample, when the image signal with the edge larger in size than thepredetermined threshold value ε is inputted, the image signal afterconversion is not shifted in phase as shown in FIG. 9, and the edge withthe steep change in pixel value is held correctly.

Further, for example, as shown in FIG. 10, even when the image signalwith the edge larger in size than the predetermined threshold value ε isinputted, the image signal after conversion is not shifted in phase asshown in FIG. 13, and the edge with the steep change in pixel value isheld correctly.

In addition, the present invention can be applied to any device handlingan image signal, such as a video camera, a digital still camera, aprinter, a display, a computer, etc.

For example, when it is applied to a computer which performs imageprocessing, a high-definition contrast correction image can be obtainedmaintaining a dynamic range in case of correcting the image contrast.When the images obtained under different lighting conditions arecomposed, only the difference of each contrast component can becorrected and a natural composite image can be generated.

Incidentally, although a series of processes as mentioned above can beperformed by means of hardware, they can also be performed by way ofsoftware. When performing the series of processes by way of software, aprogram constituting the software is installed from a recording mediumin a computer which is built into dedicated hardware, or ageneral-purpose personal computer which has various types of programsinstalled therein and is capable of performing various types offunctions as shown in FIG. 14, for example.

This personal computer 50 contains a CPU (Central Processing Unit) 51therein. The CPU 51 is connected through a bus 54 to an input/outputinterface 55. The bus 54 is connected with a ROM (Read Only Memory) 52and a RAM (Random Access Memory) 53.

The input/output interface 55 is connected with an input unit 56 made ofan input device, such as a keyboard, a mouse, a remote controller, etc.through which a user inputs an operation command, an output unit 57 foroutputting a composite picture signal to a display, a storage unit 58made of a hard disk drive etc. for storing a program and various data,and a communications unit 59 made of a modem, a LAN (Local Area Network)adapter, etc., for performing a communications process through a networkrepresented by the Internet. Further, it is connected with a drive 60for reading/writing data from/to a recording media, such as a magneticdisk 61 (including a flexible disk), an optical disc 62 (including aCD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital VersatileDisc)), a magneto-optical disc 63 (including an MD (Mini Disc)), asemiconductor memory 64, etc.

The program which causes the CPU 51 to carry out the above-mentionedfirst or second filtering process is supplied to the personal computerin the situation where it is stored in the magnetic disk 61 through thesemiconductor memory 64, and it is read by the drive 60 and installed inthe hard disk drive which is built in the storage unit 58.Alternatively, it is envisaged that it may be supplied through thenetwork. The program installed in the storage unit 58 is loaded from thestorage unit 58 to the RAM 53 according to instructions from the CPU 51corresponding to the command, from the user, inputted into the inputunit 56, and is executed.

In addition, in the present specification, steps which describe eachflowchart include processes performed in chronological order accordingto the description but also processes carried out in parallel orindividually, even if they are not necessarily processed inchronological order.

INDUSTRIAL APPLICABILITY

As described above, according to the present invention, the portionsexcept the edge can be smoothed while the edge whose change in pixelvalue is steep is held correctly.

1. A signal processing apparatus for adjusting levels of continuouslyarranged signals, said signal processing apparatus characterized bycomprising: designation means for designating the continuously arrangedsignals as a signal of attention one by one; determination means fordetermining a predetermined number of signals preceding the signal ofattention designated by the designation means, and a predeterminednumber of signals following the signal of attention, to be neighbouringsignals; weight average means for averaging by weight the signal ofattention and the plurality of neighbouring signals; flag setting meansfor calculating a difference in levels between the signal of attentionand a neighbouring signal, judging whether or not the difference islarger than a predetermined threshold value, and raising flags for theneighbouring signal and a neighbouring signal which are arrangedsymmetrically with respect to the signal of attention, when thedifference is judged to be larger than the predetermined thresholdvalue; and control means for controlling and causing the weightedaverage means to average by weight, using the signal of attentioninstead of the neighbouring signal for which the flag is raised.
 2. Thesignal processing apparatus as described in claim 1, characterized inthat said flag setting means further raises a flag for a neighboringpixel away, in view of the pixel of attention, from the neighboringpixels raised with flags.
 3. The signal processing apparatus asdescribed in claim 1, characterized in that said signals are pixelvalues of pixels constituting an image.
 4. A signal processing methodfor adjusting levels of continuously arranged signals, said signalprocessing method characterized by comprising: a designation step ofdesignating continuously arranged signals as a signal of attention oneby one; a determination step of determining a predetermined number ofsignals preceding the signal of attention designated by way of thedesignation step, and a predetermined number of signals following thesignal of attention, to be neighbouring signals; a weight average stepof averaging by weight the signal of attention and the plurality ofneighbouring signals; a flag setting step of calculating a difference inlevels between the signal of attention and a neighbouring signal,judging whether or not the difference is larger than a predeterminedthreshold value, and raising flags for the neighbouring signal and aneighbouring signal which are arranged symmetrically with respect to thesignal of attention, when the difference is judged to be larger than thepredetermined threshold value; and a control step of controlling andcausing a process in the weighted average step to average by weight,using the signal of attention instead of the neighbouring signal forwhich the flag is raised.
 5. A recording medium having a program foradjusting levels of continuously arranged signals, said programcharacterized by comprising: a designation step of designatingcontinuously arranged signals as a signal of attention one by one; adetermination step of determining a predetermined number of signalspreceding the signal of attention designated by way of the designationstep, and a predetermined number of signals following the signal ofattention, to be neighbouring signals; a weight average step ofaveraging by weight the signal of attention and the plurality ofneighbouring signals; a flag setting step of calculating a difference inlevels between the signal of attention and a neighbouring signal,judging whether or not the difference is larger than a predeterminedthreshold value, and raising flags for the neighbouring signal and aneighbouring signal which are arranged symmetrically with respect to thesignal of attention, when the difference is judged to be larger than thepredetermined threshold value; and a control step of controlling andcausing a process in the weighted average step to average by weight,using the signal of attention instead of the neighbouring signal forwhich the flag is raised.
 6. A program for adjusting levels ofcontinuously arranged signals, said program characterized by comprising:a designation step of designating continuously arranged signals as asignal of attention one by one; a determination step of determining apredetermined number of signals preceding the signal of attentiondesignated by way of the designation step, and a predetermined number ofsignals following the signal of attention, to be neighbouring signals; aweight average step of averaging by weight the signal of attention andthe plurality of neighbouring signals; a flag setting step ofcalculating a difference in levels between the signal of attention and aneighbouring signal, judging whether or not the difference is largerthan a predetermined threshold value, and raising flags for theneighbouring signal and a neighbouring signal which are arrangedsymmetrically with respect to the signal of attention, when thedifference is judged to be larger than the predetermined thresholdvalue; and a control step of controlling and causing a process in theweighted average step to average by weight, using the signal ofattention instead of the neighboring signal for which the flag israised.